Getting The Right Kind of Attention is Hard